{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Otto商品分类——Logistic 回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，分别调用\n",
    "缺省参数LogisticRegression、\n",
    "LogisticRegression + GridSearchCV （可用LogisticRegressionCV代替）进行参数调优。\n",
    "\n",
    "Otto数据集是著名电商Otto提供的一个多类商品分类问题，类别数=9. 每个样本有93维数值型特征（整数，表示某种事件发生的次数，已经进行过脱敏处理）。 竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data\n",
    "\n",
    "第一名：https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335\n",
    "第二名：http://blog.kaggle.com/2015/06/09/otto-product-classification-winners-interview-2nd-place-alexander-guschin/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1</th>\n",
       "      <th>feat_2</th>\n",
       "      <th>feat_3</th>\n",
       "      <th>feat_4</th>\n",
       "      <th>feat_5</th>\n",
       "      <th>feat_6</th>\n",
       "      <th>feat_7</th>\n",
       "      <th>feat_8</th>\n",
       "      <th>feat_9</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_85</th>\n",
       "      <th>feat_86</th>\n",
       "      <th>feat_87</th>\n",
       "      <th>feat_88</th>\n",
       "      <th>feat_89</th>\n",
       "      <th>feat_90</th>\n",
       "      <th>feat_91</th>\n",
       "      <th>feat_92</th>\n",
       "      <th>feat_93</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.018182</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.013158</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.013158</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.014286</td>\n",
       "      <td>0.315789</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.131579</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.015385</td>\n",
       "      <td>0.029851</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.018182</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.007692</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 95 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id    feat_1  feat_2  feat_3    feat_4    feat_5  feat_6    feat_7  \\\n",
       "0   1  0.016393     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "1   2  0.000000     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "2   3  0.000000     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "3   4  0.016393     0.0     0.0  0.014286  0.315789     0.1  0.131579   \n",
       "4   5  0.000000     0.0     0.0  0.000000  0.000000     0.0  0.000000   \n",
       "\n",
       "     feat_8  feat_9   ...      feat_85   feat_86   feat_87  feat_88  feat_89  \\\n",
       "0  0.000000     0.0   ...     0.018182  0.000000  0.000000      0.0      0.0   \n",
       "1  0.013158     0.0   ...     0.000000  0.000000  0.000000      0.0      0.0   \n",
       "2  0.013158     0.0   ...     0.000000  0.000000  0.000000      0.0      0.0   \n",
       "3  0.000000     0.0   ...     0.000000  0.015385  0.029851      0.0      0.0   \n",
       "4  0.000000     0.0   ...     0.018182  0.000000  0.000000      0.0      0.0   \n",
       "\n",
       "    feat_90  feat_91  feat_92  feat_93   target  \n",
       "0  0.000000      0.0      0.0      0.0  Class_1  \n",
       "1  0.000000      0.0      0.0      0.0  Class_1  \n",
       "2  0.000000      0.0      0.0      0.0  Class_1  \n",
       "3  0.000000      0.0      0.0      0.0  Class_1  \n",
       "4  0.007692      0.0      0.0      0.0  Class_1  \n",
       "\n",
       "[5 rows x 95 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "# 请自行在log(x+1)特征和tf_idf特征上尝试，并比较不同特征的结果，\n",
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"Otto_FE_train_org.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train = train['target']   \n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 默认参数的Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss of each fold is:  [ 0.79764041  0.79738554  0.79737356]\n",
      "cv logloss is: 0.797466502419\n"
     ]
    }
   ],
   "source": [
    "# 交叉验证用于评估模型性能和进行参数调优（模型选择）\n",
    "#分类任务中交叉验证缺省是采用StratifiedKFold\n",
    "#数据集比较大，采用3折交叉验证\n",
    "from sklearn.model_selection import cross_val_score\n",
    "loss = cross_val_score(lr, X_train, y_train, cv=3, scoring='neg_log_loss')\n",
    "#%timeit loss_sparse = cross_val_score(lr, X_train_sparse, y_train, cv=3, scoring='neg_log_loss')\n",
    "print ('logloss of each fold is: ',-loss)\n",
    "print ('cv logloss is:', -loss.mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression + GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "logistic回归的需要调整超参数有：C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） \n",
    "目标函数为：J =  C* sum(logloss(f(xi), yi)) + penalty \n",
    "\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同：\n",
    "1. 设置参数搜索范围\n",
    "2. 生成学习器实例（参数设置）\n",
    "3. 生成GridSearchCV的实例（参数设置）\n",
    "4. 调用GridSearchCV的fit方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': [0.1, 1, 10, 100, 1000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "#需要调优的参数\n",
    "# 请尝试将L1正则和L2正则分开，并配合合适的优化求解算法（slover）\n",
    "#tuned_parameters = {'penalty':['l1','l2'],\n",
    "#                   'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#                   }\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [ 0.1, 1, 10, 100, 1000]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)\n",
    "\n",
    "lr_penalty= LogisticRegression(solver='liblinear')\n",
    "grid= GridSearchCV(lr_penalty, tuned_parameters,cv=3, scoring='neg_log_loss',n_jobs = 4,)\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.672848371992\n",
      "{'penalty': 'l1', 'C': 100}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xd4FFX3wPHv2U0lCSGQEEooASnS\nUV7AhgooCEgRRUBAbFjA+upPFAsiKnZ9RUU62EBRiogCUkSRFpDeOwFiIi0B0vf+/thNSEJINiGb\nzSbn8zzz7OzMnZmzq+zJnXvnXjHGoJRSSuXF4u4AlFJKlXyaLJRSSuVLk4VSSql8abJQSimVL00W\nSiml8qXJQimlVL40WSillMqXJgullFL50mShlFIqX17uDqCohIaGmtq1a7s7DKWU8ijr16//1xgT\nll+5UpMsateuTVRUlLvDUEopjyIih5wpp7ehlFJK5UuThVJKqXxpslBKKZWvUtNmoZQqu1JTU4mO\njiYpKcndoZRYfn5+RERE4O3tXajjNVkopTxedHQ0QUFB1K5dGxFxdzgljjGGEydOEB0dTWRkZKHO\nobehlFIeLykpiUqVKmmiuAQRoVKlSpdV83JZshCRySISKyJbL7FfROR/IrJXRDaLyFVZ9qWLyEbH\nMs9VMSqlSg9NFHm73O/HlTWLqUDnPPbfBtRzLEOAz7PsSzTGtHAs3V0Xot26mHWcTz3v6ssopUqQ\nu79Yxd1frHJ3GB7DZcnCGLMCOJlHkR7AdGO3GqggIlVdFc+lHDi5j6/HDOaer7oV96WVUqVIYGBg\n5nrnzp2pUKEC3brl/rsydOhQWrRoQaNGjfD396dFixa0aNGCWbNmFeiaGzZs4Ndff72suJ3lzgbu\n6sCRLO+jHduOA34iEgWkAWOMMXNcFkSCFw8usrHiaAwH7jhAZHDhGn+UUirDc889x/nz5/niiy9y\n3f/pp58CcPDgQbp168bGjRsLdZ0NGzawdetWOnfO6yZO0XBnA3duN9CM47WmMaYV0B/4SETq5noC\nkSEiEiUiUXFxcYUKwqdWLYIG9efmLYYpM1/AGJP/QUoplYcOHToQFBRUqGP37NlDp06duPrqq2nX\nrh27d+8GYMaMGTRp0oTmzZtz8803k5iYyKhRo/j6668LVSspKHfWLKKBGlneRwDHAIwxGa/7RWQ5\n0BLYl/MExpjxwHiAVq1aFfpXPmLYM2yd9xOtv9nE4g4LubWu67O0Uso1XvtpG9uPxedbbvtxexln\n2i0aVSvPq7c3vuzYnDFkyBAmTpxI3bp1WblyJcOGDWPRokW89tprLF++nPDwcE6fPo2/vz+vvPIK\nW7du5aOPPnJ5XO6sWcwDBjl6RbUFzhhjjotIiIj4AohIKHAdsN2VgVgDA6j5wkvUjYE/x7+mjd1K\nKbc4ffo0q1evpnfv3rRo0YKhQ4dy7NgxAK677joGDRrExIkTsdlsxR6by2oWIvItcBMQKiLRwKuA\nN4AxZhywAOgC7AXOA/c5Dr0S+EJEbNiT2RhjjEuTBUCFbrdz7KspdF20kyldP2Fou+ddfUmllAs4\nWwPIqFHMfPgaV4ZTIMYYQkNDc23DmDBhAmvWrGH+/Pk0b96czZs3F2tsLksWxph++ew3wNBctv8F\nNHVVXJciItQdNYZ9vXqR9sWXHGjeRxu7lVLFKiQkhKpVqzJ79mx69eqFzWZjy5YtNG/enP3799O2\nbVvatGnDvHnzOHr0KEFBQSQkJBRLbPoEdxZ+DRoQcHdvOm5IZ9IPL2ljt1KqUG644QbuuusulixZ\nQkREBAsXLnT62BkzZjBu3DiaN29O48aNmT9/PgBPP/00TZs2pWnTpnTs2JEmTZrQvn17Nm3aRMuW\nLV3ewC2l5QexVatWpigmP0qPj2dbx5vZF3SewEkfc0vtW4sgOqWUK+3YsYMrr7yyQMeUxNtQrpbb\n9yQi6x29T/OkNYscrOXLU/3/htPwKCybMFIbu5UqpWY+fE2ZShSXS5NFLire0Rtboyu4/ddTTFkz\n1t3hKKWU22myyIVYLNR57S2Cz0PS+OkcPHPQ3SEppZRbabK4BP+mTfC/ozud1qXzxVxt7FZKlW2a\nLPJQ49nhmAA/rvp6A0sO/ebucJRSym00WeTBKySEqs88S5NDhkVT9MlupUqVKV3ti3KKJot8VLq7\nL7YranH7ghNMWfd5/gcopcqk4h6ifPbs2bz77ruXHbezdA7ufIjVSuSoN7H0v4eEiVM51OROapWv\n5e6wlFIlWFENUZ6WloaXV+4/07169SqaYJ2kNQsnlLvqKny7dabr6jQ+/+llbexWSuXpcoYov/76\n6xkxYgTt2rVj7NixzJ07lzZt2tCyZUtuvfVWYmNjAZg4cSJPPfUUAAMGDODJJ5/k2muvpU6dOsye\nPbvIPksGrVk4qebzL7JzyVKafR3F0uuW0KF2R3eHpJTKzS/DIWZL/uViHAPxOdNuUaUp3Dbm8uIq\ngPj4eFasWAHAqVOn6N69OyLCuHHjeP/993n77bcvOiY2NpaVK1eyZcsW+vTpU+Q1D61ZOMkrLIzw\nJ56k5X7Dz1++RmJaortDUkqVUn379s1cP3z4MLfeeitNmzblgw8+YNu2bbke07NnT0SEZs2acfTo\n0SKPSWsWBRA6YCCxM7+h+89HmXTL5wxr+4y7Q1JK5eRsDSCjRnHfz66LpZACAgIy14cOHcqLL75I\nly5d+O233xgzJvfP5+vrm7nuilvlWrMoAPH2pvbI0YSfhlOTp3Ao/pC7Q1JKlXJnzpyhevXqGGOY\nNm2a2+LQZFFAAW3b4ntre7r/lcanv7yqjd1KqYtczhDlOY0cOZJevXpx4403Eh4eXoRRFozLhigX\nkclANyDWGNMkl/0CfIx9trzzwGBjzAbHvnuBlxxFRxtj8k2nRTVEuTNSjx9nV+dORNVKo8bYT+hQ\ns0OxXFcplbvCDFFekm9DuUpJHaJ8KtA5j/23AfUcyxDgcwARqYh9CtY2QGvgVREJcWGcBeZdtSqV\nH32MNrsN877Wxm6lPNJ9P5epRHG5XJYsjDErgJN5FOkBTDd2q4EKIlIV6AQsNsacNMacAhaTd9Jx\ni9D778dWPZzuP8Ux6e/cH7xRSqnSwp1tFtWBI1neRzu2XWp7iWLx8aHWq6OodhJip0zicPxhd4ek\nlFIu485kIblsM3lsv/gEIkNEJEpEouLi4oo0OGcEtmuHz43X0euPNP63aKQ2diulSi13JotooEaW\n9xHAsTy2X8QYM94Y08oY0yosLMxlgealxkuv4oOVBt+sZtmRZW6JQSmlXM2dyWIeMEjs2gJnjDHH\ngYXArSIS4mjYvtWxrUTyqVGDSg8+yPXbDbNmamO3Up7ivl/v475f73N3GB7DZclCRL4FVgENRCRa\nRB4QkUdE5BFHkQXAfmAvMAF4DMAYcxJ4HVjnWEY5tpVYlYc8jC08lB7zYpm8cYK7w1FKuUHGEOUb\nN27kmmuuoXHjxjRr1oyZM2deVLYohigH2LBhA7/++muRxJ8flw33YYzpl89+Awy9xL7JwGRXxOUK\nFn9/arz0CpbHn2D59Ikcrt+TmuVrujsspZQblCtXjunTp1OvXj2OHTvG1VdfTadOnahQoUJmGWeH\nKM/Phg0b2Lp1K507u77DqD7BXUSCOnbEu+1/uPP3VD7+bZQ2ditVRtWvX5969eoBUK1aNSpXrkxB\nOuDs2bOHTp06cfXVV9OuXTt2794NwIwZM2jSpAnNmzfn5ptvJjExkVGjRvH1118XqlZSUDqQYBER\nEWq88hrJ3btRd8ZfLG+1nJtr3uzusJQqc95e+zY7T+7Mt1xGGWfaLRpWbMjzrZ8vcCxr164lJSWF\nunXrOn3MkCFDmDhxInXr1mXlypUMGzaMRYsW8dprr7F8+XLCw8M5ffo0/v7+vPLKK2zdupWPPvqo\nwLEVlNYsipBvnUgq3nsvN282zPhhFElpSe4OSSnlJsePH2fgwIFMmTIFi8W5n9rTp0+zevVqevfu\nTYsWLRg6dCjHjtk7g1533XUMGjSIiRMnYrPZXBl6rrRmUcQqPzqUk3PncPvcGCbdOIGhVz3u7pCU\nKlOcrQFk1CimdJ5S5DHEx8fTtWtXRo8eTdu2bZ0+zhhDaGhorm0YEyZMYM2aNcyfP5/mzZuzefPm\nogw5X1qzKGLWwAAiho+gbgwc+HIiR+KP5H+QUqrUSElJoVevXgwaNIi77rqrQMeGhIRQtWrVzGlR\nbTYbmzZtAmD//v20bduW119/nZCQEI4ePUpQUBAJCQlF/hlyo8nCBcp37YLX1S24e3kqHy173d3h\nKKWK0XfffceKFSuYOnVqZpfYgvR2mjFjBuPGjaN58+Y0btyY+fPnA/D000/TtGlTmjZtSseOHWnS\npAnt27dn06ZNtGzZ0uUN3C4bory4FecQ5c5I2rWb/b16sqg5tHjnM26qcZO7Q1Kq1CrMEOWuvA1V\nUpXUIcrLNL8G9alwT39u2Wj4avZr2titVAkzpfOUMpUoLpcmCxcKf/wJqFCebnNjmLx5krvDUUqp\nQtNk4ULW8uWp/txwGhyFXd+O50iCNnYrpTyTJgsXC+7ZA6+mjei3NJUPf3/D3eEopVShaLJwMbFY\niHh1FOXPG6rOXMHyI8vdHZJSShWYJoti4N+kMcF33cVt6w1Tf9Inu5UqCQ4NHMShgYPcHYbH0GRR\nTMKffhpLYABd58YwZYvHDKirlHJScQ9RPnv2bN59990iiz8/OtxHMfEKCaHqM88iI1/jk+/Hc+SK\n26kRVCP/A5VSHqUohyhPS0vDyyv3n+levXoVffB50JpFMapw111YG9TjnsUpfLDiTXeHo5Rygcsd\novz6669nxIgRtGvXjrFjxzJ37lzatGlDy5YtufXWW4mNjQVg4sSJPPXUUwAMGDCAJ598kmuvvZY6\ndepkDhdSlFxasxCRzsDHgBWYaIwZk2N/LeyTHIUBJ4EBxphox750YIuj6GFjTHdXxlocxGol4tXX\nSO/fn7BZv/N7s9+5scaN7g5LqVIl5s03Sd6R/xDlSTvtZZxpt/C9siFVXnyxwLEUZohysA9EuGLF\nCgBOnTpF9+7dERHGjRvH+++/z9tvv33RMbGxsaxcuZItW7bQp0+fIq95uCxZiIgV+BS4BYgG1onI\nPGPM9izF3gOmG2OmiUh74C1goGNfojGmhavic5dyV7UkqGd3bv9pHu8ueJ22D7bF1+rr7rCUUkUs\nY4jyadOmOT1EeYa+fftmrh8+fJg+ffoQExNDcnIy9evXz/WYnj17IiI0a9aMo0ePXlbsuXFlzaI1\nsNcYsx9ARGYAPYCsyaIR8LRjfRkwx4XxlBhV/vss8YsXc9ucY0xuM4lHWzzm7pCUKjWcrQFk1Chq\nfTm9yGMo7BDlGQICAjLXhw4dyosvvkiXLl347bffGDNmTK7H+Ppe+KPTFWP+ubLNojqQ9ZHlaMe2\nrDYBvR3rvYAgEankeO8nIlEislpEerowzmLnFRZGlSeepMUBw98/jCc6IdrdISmlisjlDFGemzNn\nzlC9enWMMUybNq0IIiwcVyYLyWVbznT3LHCjiPwN3AgcBdIc+2o6RkLsD3wkIhfd9BORIY6EElWQ\nBqSSIKR/fyx1azNgcQrvrdTGbqVKi8sdojynkSNH0qtXL2688UbCw8OLMNKCcdkQ5SJyDTDSGNPJ\n8f4FAGPMW5coHwjsNMZE5LJvKjDfGHPJTsglbYhyZ5xbvYbDgwfz3fUWbh75Oe0i2rk7JKU8UmGG\nKHflbaiSqqQOUb4OqCcikSLiA/QF5mUtICKhIpIRwwvYe0YhIiEi4ptRBriO7G0dpUJA2zYE3taZ\nXqsNX/wyiuT0ZHeHpFSZUevL6WUqUVwulyULY0waMAxYCOwAvjPGbBORUSKS0Q32JmCXiOwGwoGM\nkfauBKJEZBP2hu8xOXpRlRpVnn8eLy9vbv3pGFO26tj6SqmSyaXPWRhjFgALcmx7Jcv6LOCiW0vG\nmL+Apq6MraTwrlKFyo8No/UHH/DOnHHcXvd2qgfm7AeglMqPMQaR3JpKFVx+Dyl9grsEqDj4Xiw1\nIxi0MIV3V+bapKOUyoOfnx8nTpxwSZfR0sAYw4kTJ/Dz8yv0OXRsqBLA4uND9ZdfwfbQEAJnL2NF\n4xXa2K1UAURERBAdHV2gYTXKGj8/PyIiLuo/5DRNFiVE4A03END+Zu78czlvLx5Nm0E/6ZPdSjnJ\n29ubyMhId4dRqultqBKkyosv4mOsdJgXzdStU90djlJKZdJkUYL4REQQNuRhrtth+OOncRw9W/Tj\nuyilVGFosihhKj30IJZqVRi0MIX3VuU+BoxSShU3TRYljMXPj2ovjqBGnA3vuUv5I/oPd4eklFKa\nLEqiwA4dKHfdtfT9A8YuGU1Keoq7Q1JKlXGaLEogEaHKiJfwSxdunH+EqdumujskpVQZp8mihPKt\nE0noffdx8xbDsgXjOHb2mLtDUkqVYZosSrDQRx5BwkIZ9Gsy7665eBpFpZQqLposSjBLQABVhw+n\ndowNfvqNP4/+6e6QlFJllCaLEq58ly74/acV9/wOHy/Txm6llHtosijhRISqL71MuWS4/ufDTNvm\nvmkVlVJllyYLD+DXoD4VB9zDLRsNCxd+ro3dSqlip8nCQ4QNG4YlJIRBC5N5d8077g5HKVXGuDRZ\niEhnEdklIntFZHgu+2uJyBIR2Swiy0UkIsu+e0Vkj2O515VxegJr+fJUefY56kXbSF2wmJVHV7o7\nJKVUGeKyZCEiVuBT4DagEdBPRBrlKPYeMN0Y0wwYBbzlOLYi8CrQBmgNvCoiIa6K1VME9+yBb/Nm\nDPpd+HCFNnYrpYqPK2sWrYG9xpj9xpgUYAbQI0eZRsASx/qyLPs7AYuNMSeNMaeAxUBnF8bqEcRi\noerLrxB4zkbbBYeYvl0nm1dKFQ9XJovqwJEs76Md27LaBPR2rPcCgkSkkpPHlkn+TRoT0qcPt62H\n+Ys/4/jZ4+4OSSlVBrgyWeQ2c3rOCXKfBW4Ukb+BG4GjQJqTxyIiQ0QkSkSiytJ0imFPPYk1KIiB\nC1N4d502diulXM+VySIaqJHlfQSQrc+nMeaYMeYOY0xLYIRj2xlnjnWUHW+MaWWMaRUWFlbU8ZdY\nXiEhhD/9NI0OpXP210X8dfQvd4eklCrlXJks1gH1RCRSRHyAvsC8rAVEJFREMmJ4AZjsWF8I3Coi\nIY6G7Vsd25RDhbvuwufKhty3THj/jze0sVsp5VIuSxbGmDRgGPYf+R3Ad8aYbSIySkS6O4rdBOwS\nkd1AOPCG49iTwOvYE846YJRjm3IQq5WqL79Chfh0rl54QBu7lVIuJcZc1BTgkVq1amWioqLcHUax\nOzb8BU7+NJcXHi7H+Pt/ompgVXeHpJTyICKy3hjTKr9y+gS3h6v832ew+pVjwK9J2titlHIZTRYe\nzissjPAnHqfZ/nRO/baIv45pY7dSqugVOFmIiEVEyrsiGFU4If37431FXR5cauHdP98gNT3V3SEp\npUoZp5KFiHwjIuVFJADYjr1R+jnXhqacJd7eVH3pZSqeSqPlIm3sVkoVPWdrFo2MMfFAT2ABUBMY\n6LKoVIEFtG1D+S630Wu18OPvnxNzLsbdISmlShFnk4W3iHhjTxZzjTGp5PJEtXKvyv/3f3h5+dBv\nURLvrnvX3eEopUoRZ5PFF8BBIABYISK1gHhXBaUKx7tKFcIee4yrd6cTu/RXVh1b5e6QlFKlhFPJ\nwhjzP2NMdWNMF2N3CLjZxbGpQqg4+F68a9XkoSVW3l6pjd1KqaLhbAP3k44GbhGRSSKyAWjv4thU\nIVh8fKjy0kuEnUilyZIDfLnjS3eHpJQqBZy9DXW/o4H7ViAMuA8Y47Ko1GUJvOEGAjt0oM9fMPOP\nz7SxWyl12ZxNFhlDhncBphhjNpH7MOKqhAh/YTjeWOm7OIn3ot5zdzhKKQ/nbLJYLyKLsCeLhSIS\nBNhcF1bxSkkrNR8lk09EBKEPDaHt9nSif/9FG7uVUpfF2WTxADAc+I8x5jzgg/1WlMc7djqRZiMX\n0vGD390dSpGr9NCDeFWvxpDfrLy96k1t7FZKFZqzvaFs2CcgeklE3gOuNcZsdmlkxaSCVzKTvN7C\nL24Ly3bFujucImXx86PKiy9SNTaVBkv3a2O3UqrQnO0NNQZ4EvtQH9uBJ0TkLVcGVlzKpZ/lGutO\npvu8zegvf2H1/hPuDqlIBbZvT8ANN9BvpYVvV2pjt1KqcJy9DdUFuMUYM9kYMxnoDHR1XVjFKDgC\ny8MrqOBvYbr3mzw/7Tc2HTnt7qiKjIgQ/uIL+KbBXUuSeT/qfXeHpJTyQAUZdbZClvVgZw4Qkc4i\nsktE9orI8Fz21xSRZSLyt4hsFpEuju21RSRRRDY6lnEFiLPgKjfEcs8sqlrPMMHyFkMnL2NXTIJL\nL1mcfCMjqXTffdywOY39f/7C6uOr3R2SUsrDOJss3gL+FpGpIjINWA+8mdcBImIFPgVuAxoB/USk\nUY5iL2GfbrUl9jm6P8uyb58xpoVjecTJOAuvxn+w3P0V9eQIH/Mu90/8g4P/nnP5ZYtL6CMPYw2v\nzCNLrIzRxm6lVAE528D9LdAW+NGxXGOMmZHPYa2BvcaY/caYFGAG0CPnqYGMuTGCgWPOBu4S9Toi\nvb7garON19M+YNCEvzh2OtGtIRUVS0AAVZ5/nohjKdT5fR9f7fjK3SEppTxInslCRK7KWICqQDRw\nBKjm2JaX6o6yGaId27IaCQwQkWjsQ58/nmVfpOP21O8icsMl4hsiIlEiEhUXF5dPOE5qeifc9g7t\nWceTSZ8xYOJq/j2bXDTndrOg226jXOvWDPzTyvRVn/HPuX/cHZJSykN45bM/r9ZQQ97jQ+X2hHfO\nYc37AVONMe+LyDXAlyLSBDgO1DTGnBCRq4E5ItLYMeTIhZMZMx4YD9CqVauiGzK9zcNw7l96r3iH\nE/FBDJzkxYyH2hJczrvILuEOIkL4SyM436sXvZcl8X7993nnRp23WymVvzyThTHmckaWjQZqZHkf\nwcW3mR7A3rMKY8wqEfEDQo0xsUCyY/t6EdkH1AeiLiOegrn5RTgXx5D1Uzjxb3num2rhywfaEOCb\nX34t2fzq16figAG0nz6d4asWsLb+nbSu2trdYSmlSjhnn7O4I5elg4hUzuOwdUA9EYkUER/sDdjz\ncpQ5DHRwXONKwA+IE5EwRwM5IlIHqAfsL9hHu0wi0PV9aNSDF6xfEnl0HkO+jCIpNb1Yw3CF0GHD\n8KpYkUeXePHmqtGk2rSxWymVt4IM9zERuMexTACeAVaKSK7Tqxpj0oBhwEJgB/ZeT9tEZJSIdHcU\n+y/wkIhsAr4FBhtjDNAO2OzYPgt4xBhzslCf8HJYrHDHBIi8kXd9JuC7fzHDvvmb1HTPHkvKGhRE\n5Wefo/aRZCJW7uPr7V+7OySlVAkn9t/mfAqJ/AQ8aIz5x/E+HPgceBBYYYxp4tIondCqVSsTFeWi\nu1TJCTDtdtJittM38XmqN2/PB31aYLV47sC7xmbjUP97OLFvO88+4svMfvOpXC6viqJSqjQSkfXG\nmFb5lXO2ZlE7I1E4xAL1HX/tl/57GL5BcM8svEJq8FXAh+zctJqX5mzFmURbUonFQvjLL1HubCrd\nlyfqMOZKqTw5myz+EJH5InKviNyLve1hhYgEAKVnbIy8BITCwNn4+QfxQ+B7/LEuird+2enRCcO/\ncWMq3N2HTuvT2bpmAWuPr3V3SEqpEsrZZDEUmAK0AFoC04Chxphzl9ljyrNUqAkDfyTAmsbc8u/x\n44q/Gbt0r7ujuixhTz6JNag8jy714s3Vb2hjt1IqV84+wW2AP4GlwG/Y2yk890/qy1H5SuSe76lo\nO8mcCh8wfvFGJv95wN1RFZpXSAiVn36aKw4kE756L9/s+MbdISmlSiBnu872AdYCdwJ9gDUicqcr\nAyvRarRG+nxJ9ZQDzKowlrfnb+S7dUfyP66EqnDXnfg1asSDy72ZtPZTYs+Xrnk9lFKXz9nbUCOw\nz5J3rzFmEPZxn152XVgeoF5HpOc4GiRt5KuQCYz48W/mb3bv0FaFJVYr4S+/ROCZZLqt0MZupdTF\nnE0WFsdT1RlOFODY0qvZXdD5bf6TuJLPK3zNUzP+ZtlOz/yrvFzLlgT36kXXtTY2Ri1gXcw6d4ek\nlCpBnP3B/1VEForIYBEZDPyMfeA/1fYRaPccHRN/5c3g2Tzy1XqPnW2v8n+fwepfjkeXemtjt1Iq\nG2cbuJ/DPmBfM6A5MN4Y87wrA/MoN4+Aq++jT+L3PBWwmAemrmOjB8625xUaSuUnHqfh3iRC1u3R\nxm6lVCanbyUZY34wxjxjjHnaGDPblUF5nCzjSD2aPIl+fn9x7+S17IyJz//YEiakf39869Xj4WU+\nTIz6TBu7lVJA/vNZJIhIfC5Lgoh43i+hK2WOI9WOEalj6WD9mwET13LAw2bbEy8vwl96ifInk+j8\n53mds1spBeSTLIwxQcaY8rksQcaY8nkdWyZ5+ULfb5AqTXmPD2hq28GAiWs8bra9gDatKd+lCz1W\nG6I2/KyN3Uop7dFU5HyDYMAPWIIjmOj1DuGJ+xgwcQ1xCZ41217l/3sOLy8fHv7dhzfXvKmN3UqV\ncZosXMExjpTVN4AZ5d5Bzhxm4KQ1nDnvOT+43lWqEPrYozTdkUjg+t18u+Nbd4eklHIjTRauUqEm\nDJyNj0lhfoX3OB13jMFT13IuOc3dkTmt4r334lO7No8u82X8+k+JO19E85wrpTyOJgtXqnwl3PM9\n/omxLAz7mP3Rx3louufMtmfx8SF8xAhC4hLpuCqR99drY7dSZZVLk4WIdBaRXSKyV0SG57K/pogs\nE5G/RWSziHTJsu8Fx3G7RKSTK+N0qRqt4e4vCY7fzW/VvmD9vuMM+2aDx8y2F3jD9QR27MCdfxlW\nb5xPVEzxTYOulCo5XJYsHHNofwrcBjQC+olIoxzFXsI+3WpL7HN0f+Y4tpHjfWOgM/BZxpzcHqne\nLdDzc8L+XcuiWl+ydEcM//1uE+k2zxi4N3z4cLywMuR3X95Yo092K1UWubJm0RrYa4zZb4xJAWYA\nPXKUMUBGF9xgIGMkvh7ADGMnVY0uAAAgAElEQVRMsjHmALDXcT7P1awPdB5DrX+WMD/yB+ZtOspL\nc7Z4xORJPhERVHroIVpuPY/vxt3M2DnD3SEppYqZK5NFdSDruN3Rjm1ZjQQGiEg09rGmHi/AsYjI\nEBGJEpGouDgPaHxt+yjc8CyNjs9mRt3FfLv2CG/8vMMjEkalBx/Au3p1hi3zY9z6sdrYrVQZ48pk\nIblsy/mr2A+YaoyJALoAX4qIxcljMcaMN8a0Msa0CgsLu+yAi0X7l+DqwbQ9OpVxV6xh4p8H+N+S\nkj/bnsXPj/AXX6BSzHluWH2Wzj90ZvmR5e4OSylVTFyZLKKBGlneR3DhNlOGB4DvAIwxqwA/INTJ\nYz2TCHT9AK7sTufoj3mjznY+/G03kzxgtr3A9u0JuOEGBq3ypTHVeHzp4zyz/BkdP0qpMsCVyWId\nUE9EIkXEB3uD9bwcZQ4DHQBE5ErsySLOUa6viPiKSCRQD/tMfaWDxQq9J0JkO/rHjOHZyIO8Pn87\nM9cddndkeRIRwl98AUlM4oX3DvOKd09WRK+gx5wezNw5E5vxjB5eSqmCc1myMMakAcOAhcAO7L2e\ntonIKBHp7ij2X+AhEdkEfAsMNnbbsNc4tgO/AkONMZ7xcIKzvHzh7q+R8MYMjXudB2rFMvzHLfy0\nqWRXoHwjI6n11Zd4V6lCk1Gz+HZrO64KaMjoNaMZ9Msg9pza4+4QlVIuIJ7QuOqMVq1amagoD3wG\n4GwcTOmMORfHswFvM/d4MOMHXU37huHujixPtqQk4j75hJNTpuIVFsbhR27jNfmZhJQEBjcZzMPN\nHsbPy8/dYSql8iEi640xrfIrp09wu1tgGAz4EfEux7vJI7kpPJFHvtrAX/v+dXdkebL4+RH+3HPU\nnvEt1vJBVHttKtPWXsUd4bcwcctE7ph3B6uOrXJ3mEqpIqLJoiQIqQUDfsSSlsQXjKZFSAoPTYvi\n78On3B1ZvvybNaP2Dz8Q+thjJC1awt2j/mKa7yNYxMKQxUN44Y8XOJl00t1hKqUukyaLkiK8EfT/\nDmvCcb72f48aAekMnrKOHcdL/hxTFh8fwp54nMhZ3+NdpQr+I8fy6ZLaPF5zAL8e/JXuc7oze89s\nj3ieRCmVO00WJUnNNtBnOt7/bmdOpU8J9kpn4CTPmW3Pr2FDas+cQdh/nyFxxR/cOHw2M62PUrd8\nHV756xUeWPQAB86U/C7CSqmLabIoaerfCj0/xy96JQtqTAdbGgMmruGoh8y2J15ehD70EJFz5uBb\nty7poz7k9dm+jL7iSXae3Enveb35fNPnpKSnuDtUpVQBaLIoiRzjSAXuX8Ci+nOIT0rxuNn2fOvY\nu9iGjxjB+aj1NHz8c2am3k/HiA58tvEz7vzpTtb/s97dYSqlnKTJoqRq+yjc8F8q7vyWRc1WEHMm\niYGT1nD6vOf8RS4WCxUHDqDOT/Pwa96Ms2++z9DJ//DFla+Rkp7C4F8HM/KvkZxJPuPuUJVS+dBk\nUZK1fxmuupeqmz/l59Zb2B93jsFT1nHWg2bbA/uotTUnT6bq6NdJ2rGDikNeZ+qpO7jvynuZs3cO\n3ed0Z8H+BdoArlQJpsmiJBOBbh/ClbdTZ/1oZl13hC1Hz/DQNM+ZbS+DiFDhzjup8/N8Aq65hlPv\nfUiv99fxbaN3qBZQjef/eJ5Hf3uU6IRod4eqlMqFJouSzmKFOyZC7RtoFvUCX7Y7w+oDJ3jsa8+Z\nbS8r7/BwIj77lGrvvUfqocPIfc/y8eEbGH7Vc/wd+ze95vZi8tbJOsGSUiWMJgtP4O0Hfb+B8MZc\nG/U0n7dLY+nOWJ6eudFjZtvLSkQI7taVOj/PJ+iWjpz43ydc88ocfmj4HtdWu5YP139I3/l92RK3\nxd2hKqUcNFl4Cr/ycM8PUL4anTc9zjs3eDF/83FGzPaM2fZy41WpEtU/+ICIsZ+Q9m8cCYMeZcTG\n2nx03bucTj7NPQvu4c01b3I25ay7Q1WqzNNk4UkCw2DgbPDyp8/OJxlxbTlmrDvCaA+Zbe9Sgjp2\npO78+QR3786J8eOp/cQnfFf7dfo17MeMnTPoMbcHSw4tcXeYSpVpmiw8TUgte8JIPc+DB//L0P+U\nZ9KfB/h4iWcPDW4NDqbaW29SY8IEbEmJxA56kPuXwtc3TyLEN4Snlj/FE0ufIOZcjLtDVapM0mTh\nicIbQf/vkfhjPPvvSwxoUZGPftvDxD/2uzuyyxZ4w/XUmfcTIf36cnLadPwfGMGk0Kd45upnWHVs\nFT3m9ODrHV+TbvOs3mBKeTqXJgsR6Swiu0Rkr4gMz2X/hyKy0bHsFpHTWfalZ9mXc4Y95RhHSmK2\n8HrSm/RoXInRP+/g27Ule7Y9Z1gDA6jyyivU+nI6WIRj9z9E5+8O8mOHr2hZuSVj1o5hwIIB7Dy5\n092hKlVmuGzyIxGxAruBW7DPqb0O6GeM2X6J8o8DLY0x9zvenzXGBDp7PY+d/OhybZoJs4dga9id\nB84/xvI9J/m4b0u6N6/m7siKhC0xkbhPxnJyqn2SpSojR/JHrfO8ve5tziSfYWCjgTza/FHKeZdz\nd6hKeaSSMPlRa2CvMWa/MSYFmAH0yKN8P+xTq6qCaH43dHoLy855TKj4Lf+pFcIzMzeyZMc/7o6s\nSFj8/Qn/vwuTLEU/+ijNxy1nzk1f0vOKnkzdNpU75t3BH9F/uDtUpUo1VyaL6sCRLO+jHdsuIiK1\ngEhgaZbNfiISJSKrRaSn68IsBa55DK5/Bq+N0/iyzm80qlaeR78u+bPtFUTWSZbiF/xC3B39eSb+\nGqZ2noqP1YfHljzGc78/x7+JpeczK1WSuDJZSC7bLnXPqy8wyxiTtdWypqNq1B/4SETqXnQBkSGO\nhBIVFxd3+RF7sg6vwFX34vvX+8xotpHalcrx4LQoNnjAbHvOyjbJUng4R596ivDR05jRdhyPtXiM\nJYeX0H1Od77f/T0243lPtytVkrkyWUQDNbK8jwCOXaJsX3LcgjLGHHO87geWAy1zHmSMGW+MaWWM\naRUWFlYUMXuuLONIlVs6gu+vPUJYkC+DJ6/1iNn2CsKvYUNqfzeTsP8+w9nff+dIjzvod6Aqs26f\nRcOKDRm1ahT3/Xof+07vc3eoSpUarkwW64B6IhIpIj7YE8JFvZpEpAEQAqzKsi1ERHwd66HAdUCu\nDeMqiyzjSAUvfJJZHc4S4OvFwElr2B9Xup6CzjbJUp06HB/+Al7/N4ZxTUcz6tpR7Duzjzt/upNP\n/v6E5HTPmQdEqZLKZcnCGJMGDAMWAjuA74wx20RklIh0z1K0HzDDZO+WdSUQJSKbgGXAmEv1olI5\nZIwjVbkRYQse4vtuVoyBARPXEH3qvLujK3KZkyy9+CLn10Vx4Pbu3LQ+hbnd59C5dmfGbx5P73m9\nWXt8rbtDVcqjuazrbHErs11nL+VsLEzuBOdPsq/bLHrNOknFAB++e+QaKgf5uTs6l0iJjub4yy9z\nftVqyrVuTdXRrxPlFc3o1aM5knCEHnV78GyrZ6ngV8HdoSpVYpSErrPKnQIrw8A54OVH3YWD+Pqu\nasQmJDNw4lqPmm2vIDImWary+iiStm9nf/ceNFi0mx+6fs+DTR/k5/0/031Od37a95NHj6WllDto\nsijNQmrBwB8h9RxNlw5mSp9IDvx7jns9cLY9Z4kIIXfdZZ9kqW1bYse8zT+DHuCR8l2ZeftMapav\nyYt/vsiQxUM4HO/5T7srVVw0WZR24Y2h/3dw5ihtVg5hXJ/6bD16hgenrfO42fYKwjs8nIjPP6Pa\nu++ScugQB3rdQcWZS5nWcRIvtXmJrf9u5Y55dzBh8wRS03WiJaXyo8miLKjZFvpMg5gttN/4FB/1\nvpI1B07y2NcbSEkrvc8jiAjBt3ejzs/zCezYgbiPPubQ3f3oYWvK3J5zaRfRjv/9/T/6zO/DxtiN\n7g5XqRJNk0VZUb8T9PwMDqzg9n2v8maPRvbZ9r7zzNn2CsKrUiUiPvwwc5KlA3f1gS++4b1r3mJs\n+7GcTT3LoF8G8fqq14lPKV3PpChVVDRZlCXN+0KnN2H7XPrFfcyI2xry8+bjvPDjZmylPGFAjkmW\nvviCA3f0pvWJYOb2mMuARgOYtWcWPeb0YOHBhdoArlQOmizKmmuGwvXPwPopPJT+LU90qMd3UdG8\n/vP2MvEDmW2SpcTzHOzXn4T3PubZxsP4pus3hPmH8ezvzzJs6TCOnb3UgANKlT2aLMqiDq/AVYNg\nxbs8HbSE+6+LZMrKg3z4m2fPtlcQOSdZ2t+jJ7X3nOWbrt/wbKtnWRezjp5zezJt2zTSbKWz55hS\nBaHJoiwSga4fQsNuyK/DebnWVvq0iuB/S/YwYYXnz7bnrIxJlmpOnwYW4fDgwcS9NpoBtXozp8cc\n/lPlP7wX9R79f+7PthPb3B2uUm6lyaKssnpB70lQ+wZkzqO81SyWrk2r8saCHXyzpmw9fxDQujV1\n5syh4v33c/r779nf7XbKr9/D2PZjee/G94hLjKP/z/15e+3bnE8tfUOmKOUMTRZlWZZxpKzfD+Kj\n61K4uUEYI+ZsYe7Go+6Orlhlm2QpKJAjDz/C8eHD6VihDXN7zuXOenfy1Y6v6Dm3J78f+d3d4SpV\n7DRZlHV+5WHADxBUBe8ZdzOuUzla167IM99tYvH20jHbXkFknWTpzM8L2Nftdli2mpeveZkvb/uS\nAO8Ahi0dxjPLnyH2fKy7w1Wq2GiyUI5xpGaDlx++397F5F5VaFI9mKHfbGDl3rI381y2SZYqV+bo\nk08S/cSTNLFE8F2373ii5RP8fuR3eszpwcydM3WiJVUmaLJQdiG1M8eRCph5J9PvjiSyUgAPTY9i\n/aHSM9teQWROsvTMM5xdvpz9Xbtxfv4CHmz6ILN7zKZxaGNGrxnNoF8GsedU2elJpsomTRbqgvDG\n0G8mnIkm+Mf+fDWoEZWDfLlvylq2HTvj7ujcQry8CB3yEJFzZuNTpw7Hnh/OkUceoeo5bybcMoE3\nr3+Tw/GH6fNTHz7e8DFJaUnuDlkpl9BkobKrdQ30mQ7HNxM2/36+GtyCQF8vBk1ay75SNtteQfjW\nqXNhkqW169jf7XZOz/yObpFdmdtzLl3rdGXiloncMe8OVh1blf8JlfIwLk0WItJZRHaJyF4RGZ7L\n/g9FZKNj2S0ip7Psu1dE9jiWe10Zp8qhfifo8Skc+J2IZU/y1f2tECm9s+05S6xWKg4aSJ15c/Fr\n1pSYkSM5fN/9BMQmMPr60Uy6dRIWsTBk8RBe+OMFTiaddHfIShUZl82UJyJWYDdwCxCNfU7ufpea\nHlVEHgdaGmPuF5GKQBTQCjDAeuBqY8wlb57rTHku8NdYWDQCrr6P7Ve9Rt8JqwkJ8OH7h6+hcvnS\nOdues4wxnJ41i9i338GkpRH21JNUHDiQFNKYsHkCk7ZOIsA7gP9e/V96XtETEXF3yErlqiTMlNca\n2GuM2W+MSQFmAD3yKN8P+Nax3glYbIw56UgQi4HOLoxV5ebaYXD907B+Co12jWXq/a2JS0hm4KS1\nbD16plTPh5GfzEmW5v+UOcnSof73wMFohrUcxqzbZ1E3uC6v/PUKDyx6gGWHl3Eo/pAOHaI8lpcL\nz10dOJLlfTTQJreCIlILiASW5nFsdRfEqPLT4VU49y+seIerAkKZOKg390xaQ7dP/sQiULtSAPXD\ng6hfJYgG4UE0qBJIrUoBeFvLRnOYd5UqRHz+GfHzf+afN97gQM9ehA4dSp0H7mdK5yn8uOdHRq8e\nzbqYdfbyFm9qla9FZHAktcvXJjI4kjrBdYgMjqScdzk3fxqlLs2VySK3evel7nn1BWYZYzL+VHXq\nWBEZAgwBqFmzZmFiVPkRgW4fQeIp+OX/uLZ3JVY+35UNh0+x+5+z7I5JYPc/CSzaHkPGKOc+Vgt1\nwuxJpEGVIPtreBARIf5YLKXvdkzGJEsB115DzOjRxH30EfGLFlLtjTe488o76RLZhb2n97L/zH4O\nnDnAgTMH2HNqD0sPLyXdXKidhZcLJzI4MttSJ7gOYf5hehtLuZ0r2yyuAUYaYzo53r8AYIx5K5ey\nfwNDjTF/Od73A24yxjzseP8FsNwY823OYzNom4WLpSbBV73hyGp799p6HbPtTkpNZ2/sWXb/k2BP\nIv8ksCsmgaOnEzPL+HtbqRcemJk8Mmoj4eV9S9WPYfzixcSMGkX6qdNUevABQh97DIuPz0XlUtNT\nOZJwxJ5A4u1JZP/p/RyIP8C51HOZ5QK8A4gsnz2BRAZHUiOoBt5W7+L8aKoUcrbNwpXJwgt7A3cH\n4Cj2Bu7+xphtOco1ABYCkcYRjKOBez1wlaPYBuwN3JfsXqLJohgknYGpXeGfbRBcA3qMhaot7EOG\nXEJCUip7YjNqII4k8k8CcQnJmWWC/LyyJY/64UHUDw+kUqBvcXwql0g/c4Z/xrzNmdmz8albl0r3\nDca34ZX41rsCi2/en8sYQ1xiXGYtJGuN5J/zF4ZgsYqVGkE1qB1cOzOBZCzlfS7930SprNyeLBxB\ndAE+AqzAZGPMGyIyCogyxsxzlBkJ+Bljhuc49n7gRcfbN4wxU/K6liaLYnI2Fv7XElIynrkQCK0H\n1VpCtaug+lVQpSl4++d5mpPnUhy1EMcSc5adMfHEJ11oAA4N9HEkjgu3s+qHBxLk5zl/TZ/940+i\nhw7FpKTYN1it+ETWxq9+A3wbNsSvYQN8GzTEq7Jzt5rOpZ7j4JmDmQnkYPzBzNesjeeh/qH2xJGj\nRhIeEI5FykZ7knJOiUgWxUmTRTE7dwKO/e1YNsDRDXA2xr5PrFC5EVR3JJBqLe1Ph+dzy8QYQ2xC\nMrtiLiSRXf+cZc8/CZxPuXBvv1qwX7ZaSIMqQVxRORA/b6srP3GhmfR0Ug4fJnnXbpJ27SR55y6S\ndu0k7djxzDLWChXsyaOBI4k0qI/PFVfkevsqN2m2NI6ePXpRbWT/mf0kpCRklvP38qd2+drUDs7e\nuF6rfC18rZ5bk1OFp8lCFb/44xcSR0YSSXQ8GmP1tdc4qrW01z6qXWWvkVjy/4G32QxHTyeyK8Z+\nC2uPI4nsiz1LSrp9ED8RqFWxXPZG9SpBRIaW3J5Z6fHxJO/aRZIjeSTv2k3y7t2YZMctOi8vfCMj\nM5OHbwN7TcQrLMzpaxhjOJl0MvvtrPgDHDxzkKNnLwxDLwjVA6tnSyAZS4hfSFF/dFWCaLJQ7mcM\nnDqYpfbxNxzfeOEWlk8gVG3uuIXlSCIhkfZffiekpds4eOJ8tttZu2ISOHjiPOmOrlneViEyNOCi\nRvUaFcthLYE9s0x6OimHDpG8cydJu3Y7XneRFhOTWcZaqVK25OHbsCG+kZGIk7WQDIlpiRyKP5RZ\nG8lYDsYfJDn9QptSBd8KFyWQyPKRVAushtWJZK9KNk0WqmSypcOJvY7ah6MGcnwzZPw4+VXIXvuo\n1hLKV3M6gYC9Z9b+uHOZjel7HK9HTl7omeXnbeGKyhf3zKoa7Fcie2alnTpF8u49JO/aSdLOXSTv\n3Eny3r0X2kK8vfGtUyezDcS3QX38GjbEq1KlAl8r3ZbO8XPHL7qldTD+YLYhTHwsPtQKrnVRu0it\n8rX0mREPoslCeY70VIjdbk8cGUnkn+2Q8QxCYPiFxFHd8RoQWuDLnEtOy+yZtStLbeSf+Cw9s3y9\nqBcemO35kPpVgggtgT2zTFoaKQcP2pPHLnsNJHnnLtJiL0zKZA0LxS9L8vBt0MBeC/EuXCeB00mn\nORh/MFsPrQNnDhB9NjrbvB5VA6pe1NU3MjiSSn6VSmQyLss0WSjPlpoIMVsv1D6OboB/d5P5bGZw\nzewN6NVagF9woS51+nwKu/85e6EW4kgmp8+nZpapFOCT2RsroxZSLzyIYP+S1zMr7dSpzNtX9sb0\nXaTs3YtJtX8e8fbG54orHI3pDTKTiFdI4dsmktOTORx/+EICibc/M3Iw/iCJaRdqdEHeQRc9eBgZ\nHElEUATelpL3XZYFmixU6ZMUDzGbs9/COnXwwv5K9bLfwqrSFHwKdzvEGEPc2WR2x5zNditrd0wC\n57L0zKoa7HchiWTpmVXOx5WDIxScSU0l+cABR4O6o0fW7l2kx12YCdGrcmV78mhgTx5+DRvgU7s2\n4lX4z2IzNmLPx15UEzlw5gBxiXEXrm3xomZQTSKDI9n27za8rd4MbjwYb4s33lZvfCw++Fh98LZ4\nZ75mbM+6Les+L/HSWowTNFmosuH8yQuN5xkN6QmOLqlihcpXZm9Ar9wYvArWEJyVMfaeWfbG9LOZ\nSWRP7FlS0i70zKpZsRz1KtvHyqofHkTFAB+sIlgsgtUiWAQskrFuf822LoLFQua65LI9e9nC/Sim\nnTiRmTySd9t7ZiXv3w8ZtRBfX3yvuCIzefg2sPfMslaoUOjvMENCSsJFCSSjp5a55MhAzhMkWxLx\nsng5lXQyt18iUXlZvOzlchyTcX5njilJHQM0WaiyK/549uc/snXh9bnQhTfjIcLQ+k514c1Lus1w\n6MS5zCSyO9ZeC9n/77nMnlmuli2ZiD2ZWLIlIbIlrMx1x6tF7OfwtqVT5XQMVU9EU/XfI1T59wjh\ncUcIOH/heY2EoIqcqFKTf8NrcapKLU5VqcnZsGpYrFbH+cl2fqslyzVyXDd7LDDlr30YSaJf2+qk\nm1TSTRrpJpU0k4rNpJFmUi9st6Vk7s9a9lLbMo696Dx5HJNuUrFRtKMFCxas4o1VvHK8OtbxxpJz\nW8517K9rj21B8GLzo5ccDSnvWDRZKOVgDJw+lOX21Ub7kvGwmneAvQtvRuN5tZZQsU6BemBdSnJa\nOgf+PUdCUhrpNoPNZkg3hnSbwRh7kkk32bfbjCHdxkXb7O9zbL/oeBzHZz0m+/kuvtaF42w5ttts\n2Mum2yh37gzhcUeoGneEqieOUO3EUcJPx2B1NGwnW705WqEqh0Kqc7hCNQ4EV+Ng+aokePtfiMfx\nmT2PDcQGkoZIOkhajvX0HOtpF9Yt9tdLl8/tnFnKW9KAnOfJWE9HxIYtqSrbHl5UqE+myUKpvNhs\ncGJP9h5YMVsgYw5tv+DstY9qLaF89SJJIKWJLSWFlL17HT2ydjka1XeSfjpz0ku8q1WzPwvSoD5+\njmdDrBE1sInknpgKmEwK8p+kIP/1CtLeUbDzFqCsE2dON+kY0qkUEFCAKLLGo8lCqYJJT4XYHdl7\nYMVuh4wxlwIqZ6l9OJJIIbrwlnbGGNJiYzOfTs/omZVy4IA9SQPi749v/Xr2xvSMHln162MNDHRz\n9GWPJgulikJqon2U3aw9sOJ2caELb43sz39Ua1noLrylnS0pieS9+7I9E5K0axe2M2cyy3hHRGA7\nexbx8SG4R3fE3x+Lnz8Wfz8k89XPvq2cv33d3x+Ln5+jrN9l9d4qizRZKOUqyQlwfFP2W1jZuvBe\ncfEovD6Fu0VQ2hljSIuJyZI8dnJ22XJMaipisWQ+G1Ig3t5ZEogjsWRJJvYkk2N/ZjLKJTFlnqsc\nFn8/LH5+4O1darrlarJQqjidP5l9DKxjf0PCsQv7LV5QLtTebdeasXjbB1jMWPfydWzzcWz3vlDW\nK69jfHI/Ltu1clw347gS1IUzNyYtDVtSMiYpEVtSEiYxEVtiIrbEJPu2xCRsSYmYpCRs5xPt64lJ\n9rJZ95+3H59tf6LjnElJBQ/Mas2egPz9L9RssiaocjlqRln356wZZU1Q5cohPj7FkpCcTRZaX1Oq\nKJSrCFd0sC8ZMrrw/vJ/9naPerfa20XSkyE9xb6elmx/TTkHiScd+1MgLcVRJiX7MUVNLHkkrEsk\nmDyTXI4l14SVkcwukRy9fO1xYW9ktnoBgT72RYLJbE7O/CGV7OtO7buwbmw2THJytgRiS0zCJJ53\nrDuSUdYElZiYPRk5EpAt8Ty2f0+QmvVcjnUK+oe5yIVklEdiOr9qNeLnxxWLC9cbylmaLJRylfJV\n7UvDLkVzPmPsSSctOUcSyUgwWdZzLmm5bMtMVilZjku+RMJKgcTzObbncoytaJ9HKA4ZqcOSbQvO\nJRxvwEegfC77spzDGMHYBFsamHSL49X+3pYumDTBliaOdbCliX1/agK29ATHfrDFC+YkpKdBWpr9\n6zYpNrxsF8blchWXJgsR6Qx8jH2mvInGmDG5lOkDjMTeYrjJGNPfsT0d2OIodtgY092VsSpV4ok4\n/vouwWMo2dJzT1iXqillrV1lJJ6VnwAGWj9kP2fmX+Qm+3rmPnNxuYv2cWFfoc+RXxw595G5TzD2\nhFTgczgR4+6F4OX6UX5dlixExAp8CtwCRAPrRGSeMWZ7ljL1gBeA64wxp0SkcpZTJBpjWrgqPqWU\nC1is9sXbr/DnaHV/0cWjiowrpxBrDew1xuw3xqQAM4AeOco8BHxqjDkFYIyJRSmlVInjymRRHTiS\n5X20Y1tW9YH6IrJSRFY7bltl8BORKMf2ni6MUymlVD5c2WaRW5+vnN0BvIB6wE1ABPCHiDQxxpwG\nahpjjolIHWCpiGwxxuzLdgGRIcAQgJo1axZ1/EoppRxcWbOIBmpkeR8BHMulzFxjTKox5gCwC3vy\nwBhzzPG6H1gOtMx5AWPMeGNMK2NMq7ACTGKvlFKqYFyZLNYB9UQkUkR8gL7AvBxl5gA3A4hIKPbb\nUvtFJEREfLNsvw7YjlJKKbdw2W0oY0yaiAwDFmLvOjvZGLNNREYBUcaYeY59t4rIdiAdeM4Yc0JE\nrgW+EBEb9oQ2JmsvKqWUUsVLh/tQSqkyzNnhPlx5G0oppVQpoclCKaVUvkrNbSgRiQMOXcYpQoF/\niyicoqRxFYzGVTAaV8GUxrhqGWPy7U5aapLF5RKRKGfu2xU3jatgNK6C0bgKpizHpbehlFJK5UuT\nhVJKqXxpsrhgvLsDuASNq2A0roLRuAqmzMalbRZKKaXypTULpZRS+SqzyUJE7hKRbSJiE5FL9iIQ\nkc4isktE9orI8GKIq+tTU2AAAAZISURBVKKILBaRPY7XkEuUSxeRjY4l55hbRRlPnp9fRHxFZKZj\n/xoRqe2qWAoQ02ARicvy/Tzo6pgc150sIrEisvUS+0VE/ueIe7OIXFVC4rpJRM5k+b5eKaa4aojI\nMhHZ4fi3+GQuZYr9O3MyrmL/zkTET0TWisgmR1yv5VLGdf8ejTFlcgGuBBpgH9G21SXKWIF9QB3A\nB9gENHJxXO8Awx3rw4G3L1HubDF8R/l+fuAxYJxjvS8wswTENBgY64b/p9oBVwFbL7G/C/AL9uH7\n2wJrSkhcNwHz3fB9VQWucqwHAbtz+W9Z7N+Zk3EV+3fm+A4CHevewBqgbY4yLvv3WGZrFsaYHcaY\nXfkUc2a2v6LWA5jmWJ8GuHPiJ2c+f9Z4ZwEdRCS3uUyKMya3MMasAE7mUaQHMN3YrQYqiEjVEhCX\nWxhjjhtjNjjWE4AdXDxBWrF/Z07GVewc38FZx1tvx5Kz0dll/x7LbLJwkjOz/RW1cGPMcbD/TwtU\nvkS54phJ0JnPn1nGGJMGnAEquSgeZ2MC6O24bTFLRGrkst8d3PH/k7Oucdze+EVEGhf3xR23S1pi\n/2s5K7d+Z3nEBW74zkTEKiIbgVhgsTHmkt9XUf97dOVMeW4nIr8BVXLZNcIYM9eZU+Sy7bK7j+UV\nVwFOk+9MgkXAmc/vku8oD85c7yfgW2NMsog8gv0vrfYujMlZxf1dOWsD9iEfzor8f3v3E2JVGcZx\n/PuzsoKJohkpgyCqgUBKF5WhswlcxCyMQphFkAtbuGjRKgjCqBYRQbsCydpFC/MPogMt1BpahAVZ\nMzJB2iICNREyAivRp8X73LiNc++5c73n3El+Hxg8c+ece5/7cs99PO8553k0SekzM97Ui0saAfYA\nL0XE7wv/vMgmjYxZRVxDGbOIuAysk3QHsE+ls2j7uajaxuu6ThYRsekan6KXbn9L1i0uSWclrY6I\n03m4/WuH5/i3k6Ckzyn/+xl0sui12+G9wC+SbgRup94pj8qYIuJ8268fAG/XGM9S1PJ5ulbtX4QR\nMS3pfUljEVF7DSRJN1G+kD+OiL2LrDKUMauKa5hjlq/5W+73TwHtyaK2/dHTUN310u1v0A4AW3N5\nK3DVEZCa6yTYy/tvj3cLcCTy7FpNKmNaMKe9mTLnvBwcAJ7PK3yeAC60phyHSdLdrXltSY9TvhfO\nd99qIK8r4ENgPiLe7bBa42PWS1zDGDNJq/KIAkm3ApuAHxasVt/+2OTZ/OX0AzxDycJ/AWeBz/Lx\ne4DptvUmKVdDnKJMX9Ud1yhwGPgx/70zH38U2JXLG4BZypVAs8C2GuO56v0DbwCbc/kWYDdwEjgG\n3N/AGFXF9BZwIsfnKPBQQ5+pT4DTwKX8bG0DtgPb8+8C3su4Z+lwFd4Q4nqxbby+AjY0FNcEZYrk\ne+B4/kwOe8x6jKvxMQMeAb7NuOaAHfl4I/uj7+A2M7NKnoYyM7NKThZmZlbJycLMzCo5WZiZWSUn\nCzMzq+RkYbYEkv6oXqvr9p/mXfdIGpG0U9KprCI6I2m9pJW5fF3fNGv/L04WZg3J+kE3RMRP+dAu\nyt214xGxhlItdyxKgcTDwNRQAjVbhJOFWR/yjuJ3JM1JmpU0lY+vyNIPJyQdlDQtaUtu9hx5R76k\nB4D1wKsRcQVK6ZaIOJTr7s/1zZYFH+aa9edZYB2wFhgDvpY0Qym9ch/wMKVi8DzwUW6zkXI3NcAa\n4HiUwnCLmQMeqyVysz74yMKsPxOUyraXI+Is8AXly30C2B0RVyLiDKXcSMtq4FwvT55J5G9Jtw04\nbrO+OFmY9adTQ5lujWYuUmr3QKkrtFZSt33wZuDPPmIzGzgnC7P+zABT2YxmFaV16THgS0rjpRWS\n7qK032yZBx4EiNJ75Bvg9bbqpeOSns7lUeBcRFxq6g2ZdeNkYdaffZTqn98BR4CXc9ppD6Wy6xyw\nk9Jh7UJuc4j/Jo8XKE2wTkqapfTeaPVqeBKYrvctmPXOVWfNBkzSSJQOaqOUo42NEXEmexAczd87\nndhuPcde4JWo7hNv1ghfDWU2eAezSc1K4M084iAiLkp6jdIn+edOG2dTp/1OFLac+MjCzMwq+ZyF\nmZlVcrIwM7NKThZmZlbJycLMzCo5WZiZWSUnCzMzq/QPZZBNIsiBtKYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111c39d90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(Cs)\n",
    "number_penaltys = len(penaltys)\n",
    "test_scores = np.array(test_means).reshape(n_Cs,number_penaltys)\n",
    "train_scores = np.array(train_means).reshape(n_Cs,number_penaltys)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs,number_penaltys)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs,number_penaltys)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(penaltys):\n",
    "    #pyplot.plot(log(Cs), test_scores[i], label= 'penalty:'   + str(value))\n",
    "    plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = penaltys[i] +' Test')\n",
    "    plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = penaltys[i] +' Train')\n",
    "    \n",
    "plt.legend()\n",
    "plt.xlabel( 'log(C)' )                                                                                                      \n",
    "plt.ylabel( 'logloss' )\n",
    "plt.savefig('LogisticGridSearchCV_C.png' )\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图给出了L1正则和L2正则下、不同正则参数C对应的模型在训练集上测试集上的logloss。\n",
    "可以看出在训练集上C越大（正则越少）的模型性能越好；\n",
    "但在测试集上当C=100时性能最好（L1正则）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型，用于后续测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import cPickle\n",
    "\n",
    "cPickle.dump(grid.best_estimator_, open(\"Otto_L1_org.pkl\", 'wb'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
